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Abstract 



Even though several national testing firms have developed measures to 
evaluate the effectiveness of baccalaureate education, there continues to. 
be a general reluctance on the part of faculty in colleges and universities 
to aq£ept these measures as criteria on which- to evaluate educational C~ 
programs. Some' of the resistance appears to lie in the lack of validity of 
corrniercially prepared tests as measures of their competencies (often 50 to 
100 in rfumber), the amount of cost and effort to evaluate these competen- 
cies with the kinds of performance tests they seem to prefer, and question- 
nable psychometric rigor of ' locally made performance tests. A' series of 
throe prototypical real-life problem solving exercises and. rating criteria 
were developed to address' these limitations. A* theory of Tiumart performance 
provides a rationale for the development of the generic skills test, the 
stimuli are 'in the formjGt£ real -life scenarios, and require short-answfer 
essay responses. Rating scales use descriptions of high, medium and/low 
performance attributes to evaluate responses. /The results of a field test 
indicate that the difficulty level of exercises appears acceptable, the 
problenr-solving constructs are' independent and meaningful, and that the 
interrater and alpha reliabilities are sufficiently high to warrant its use 
as a program evaluation instrument. The contribution of basic intellectual 
factors J maturation and educational experiences to generic performance 
should be ascertained through further research. ' 
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The Development of a Baccalaureate Outcome Measure 

■ j 

Based on a Generic Skills Theory "of tjiiman .Performance 

b y - • * . 

6ary W. Peterson 
Florida State University 

*,-)!* ' . 

\ ' 

While several showcase tompetency-based liberal arts programs 
have been implemented i,n higher education and several national testing 
firms are 'either developing or marketing instruments* to measure^ bac- ^ , 
calaiireate outcomes (Forrest and Steele (1981), Warren, (1980) , Winter, 
McClelland and Stewart 1981), the nature of the baccalaureate outcomes and 
their measurement is still very much open to question. Further* the press 
for accountability in higher education (Peterson and Stakenas, 1981), has 
resulted in many instituiions either trying purchase outcore measures from 
co^nerCial vendors or trying to develop their own in order to demonstrate 
"value -added" contribution's to intellectual skills (Katz, 1982, Astin 
1982). In spite of the existence of.jnodel competency .programs and commer- 
cially prepared instruments faculty often reject available tests out of. 
hand at face value or fall short in their efforts to create them (Peterson 
and Watkins', 1979, Peterson, 1982). , • 

There a^e additional deterants that discourage faculty from- using com 

■ - - - • • ■ - ~~; * 

jjetency outcome measures to evaluate the effectiveness of baccalaureate 
programs. First, faculty, in the process of defining desired bac- 
calaureate outcomes, often formulate ponderous lists of 50 to 100 com- 
petency statements. But, without a clear and coherent theoretical fra- 
mework with which to abstract such lists, the selecting or developing of .- 



assessments is confusing, if not overwhelming ., Secondly, while faculty are 
generally favorably disposed to the use of free response questions as a ; 
strategy for assessing higher order thinking skills. (Peterson, and Watkinsj 
1979), the amount' of time,' inconvenience and resources required to admi- ', 
ninster and score such instruments alscpresent drawbacks— not to mention . 
certain methodologiic^prolJlems'. jpiirdly, the task of developing a bac- 
calaureate test^tnat takes into account differences among cohorts. of stu- 
dents in terms of Intel lectuaT factors as well as "educational, and life's' 
experiences also presents formidable psychometric challenges. A prototype 
baccalaureate outcome measure was developed from a -comprehensible theoreti- 
cal framework that offers practical utility and psychometric integrity. 

Based on earlier theoretical .research (Peterson 1981, 1982, Petersqn 
and Rumsey, 1981), the prototype baccalaureate measure was derj^vied from a { 
model of human performance that links baccalaureate'skills dynam<}cally (see 
Travers, i980) so that multiple skill dimensions can be assessed through 
the use of a" single stimulus situation (i.e., scenario). To assist \faculty 

-in developing locally-made versions of the instrument, a framework for x the 
development of scenarios and their respective stimulus and response attri- 

■ butes is also presented. Analyses pertaining to construct validity^ \ 
reliability and'utility are included. 

... , 

A generic skills theory- of human performance V . - 

The development of the prototype baccalaureate assessment battery 
began with the concept of c ompetence . Competence may be considered as the 
integration of knowledge, skills, arid attitudes required to perform adult 



roles effectively (McClelland, 1973; and Goleman, 198L). A competent 
indiv^* jual , then this one who.is able to meet effectively and effi- 

1 . • ■* ■ ; 

ciehtly the demands of occupational, personal, and community roles as in 

i K ~ • . • 

the case of a competent physician, army officer, or housewife. Competent 
performance* is based on the mastery and integration of .component cognitive 
~^fod psychomotor skills and knowledge. These component skills are typi- 

cally labeled competencies , > 

» * -t 

Some ski 1 Is, t however, seem to occur over andjover again ^in the analysis 
of many and diverse occupational or life tasks. These pervasive and 
transcendant skills are designated as generic skills . According te Wodit^h 
(1977), the term, generic,, connotes a function or a pattern of activity . * 
that is recurrent in a wide Series of discrete purposive behaviors. 
-"Generic skills are basic in the sense that they are ubiquitious: they 
show up again and again as components or instances of successful behavior" 

j : . 

(pg. 8). Thus one infers the presence of generic skills from the' success- 
ful performance of discrete tasks which constitute. diverse occupational or 
life roles. A task is defined as a unit of behavior with a^beginning and 
ending point that can be accomplished J n hours rather than days or weeks 
(Branson, 1980). 

Important attributes of; generic skills may include the following 
(Peterson, 1981): ^ 

• Generic skills undergrid what is commonly referred to as genera- 
lized problem-solving behavior; 

• A generic skill is an ability or capability that possesses it's own 
unique hierStrchy/of discrete related component skills; 

• A generic skill is pervasive and recurs across academic or pro- 
fessidonal disciplines of study and even life or job skills; ' 



• • # The mastery of knowledge base underlies the development and 

demonstration .of t generic skills;* t v 4 

» ■ • 

9 The demonstration of generic skills requires the Integration of 
discrete lower order component skills and knowledge; and 

' 0 Individuals who have mastered generic skills are able to apply 
them in a variety of real-life situations or^contexts tfo solve . 
problems encountered in. adult roles jn society. ^ 

% . Thus, generic skills are constructs which could be viewed as "common 
denominators'! of learning among various disciplines of study and ms(y be 
used to compare student development across academic programs within a uni- 
versity or even among colleges and universities. Stated in more formal 
terms, generic skills are constructs of intellectual^abi lity that capture 

'common variance across- discretely different discipTine-oriented. performance 

. ' ' I" ' 

tasks. * 

e * . . - * K 

Structural and functional relationships among generic skills " M 

Four generic skills have been identified from previous research by 
Peterson (1982). and Warrea (1980) for the development of measures: they - 
include Communication, Disciplined Inquiry, Valuing, and Execution. These 
skills undergird a capstone skill referred to. as generalized 
Problem-solving. Structurally, according to trfe author's formulations (see 
figure 1, each/of these skills in turn possesses its own unique hierarchy 
of 1 subordinate skills such as the perceiving and interpreting of verbal, 
non-verbal and quantitative symbols in Communications or analyzing or 
synthesizing in Disciplined Inquiry. Further, the capacity to demonstrate 
generic skills is based on a set of basic cognitive abilities (e.g., rela- 
tionships among words and concepts, proportionate logic, Aspac'ial / 
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Figure 1. An hypothetical structure and relationships among problem 
solving,. generic, and basic cognitive skills.' . . 



relationships) that are typically measured by common intelligence and apti- 

tude. tests. • ''•'•'•>* 

In terms of the functional characteristics. of the model, the^trst « 
step in coping with problem situations (anomalies, needs, dissonances) is 
to perceive and interact with the phenomenon of inquiry (i.e. a problem 
solving task). This, calls for receiving, interpreting, and sending verbal, 
nonverbal, and quantitative messages (Communication). Individuals must 
then be able to inquire into the phenomenon by reducing it into its com- 
ponent parts, and relating them to each other • (Analysis) . They must then 
extract and organize information to formulate potential coyrses of action 
to reduce or eliminate the anomaly (Synthesis). Next, each alternative 

* ™ 

course of action must be evaluated in terms of costs and benefits to them- 
selves, their social or occupational groups, and to the wider society in 
terms of higher order principles £r ethics (Valuing) . Finally, individuals) 
must structure activities to achieve a goal by defining objectives in a . • 
coherent means-ends relationships (Planning) and carry' out the plan 
(Implementation).' 

'All of these capabilities are brought to bear in performing essential 
life and occupational tasks in which the cue functions are ambiguous, the 
alternative courses of action not predetermined, and the 'consequences of ; 
implementing various courses of action uncertain. If there is a lack bf 
development in any of these capabilities, a person's potential as a 
problem-solver and hence performer of complex life tasks is limited con- 

r *.'. - 

r* • 

comitantly. The portrayal of this process may be depicted in Figure 2 on 
the next page in the f()rm of^an N 2 chart. One can note from the chart how 0 " 
successive skills convolute on preceding skills. The engaging of a latter 



7 




Where : 



COM is perceiving a problem or need (Cormiunication) 

ANAL is relating components (Analysis) 

SYN is formulating possible courses of action (Synthesis) 

VAL is estimating costs and benefits for each course of action (Valuing) 

EXEC is planning and implementing a solution (Execution) 



Figu7# 2 



N 2 Chart for Interrelationships Among Subordinate Generic 
Problem Solving Skills 



»» g . — 
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Figure 3. A model for the relationship among skill, content, and scope 
1n generic skill test.develonment, >, 
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\ • 

a narrow or discrete sphere of knowledge, a mastery of several operations 
and an ability to apply these skills 1n .familiar Instructional problem 
situations. Thus the cue function (I.e. test Item stimulus), should be 
recognizable to students who. have had the dourse and the responses may 

require only the retrieval of knowledge and skills practiced 1n familiar 

*■ * 

course contexts. * * 

A second or tnore distal level of outcome, I.e. the major or minor 
field, concerns whether students are able <to demonstrate generic skills 
using a wider range of knowledge and more .complex operations and procedures 
than at the proximal level. Students must be able to Integrate knowledge 
and skills from a series of related courses and use specialized, knowledge 
to' address more complex situations than at the course level of outjjpme. 
Assessment situations a'c the second level also Involve more complex cue . 

• • • 

functions than at the proximal level.. Students must be able to perceive 
order 1n more ambiguous stimulus situations by being able to to dlfferen- 
J1ate relevant from irrelevant information. They must bring to bear more 
complex procedures than at the course level, select appropriate operations, 
according to their utility end efficiency, and integrate them to address 
higher order purposes. The responses may also be more sophisticated 
requiring more" complex and ordered sequences of expression while using 
appropriate information to document and to justify. These capabilities are 
^performed through the "gestalt" of a discipline of study; 

Assessment of generic skills at the third level, i.e., the B.A, level, 
requires students to investigate issues or problems integrating^both speci- 
fic-major field and broad general education perspectives. A student should 

be able to not only use formal structures from one's discipline of study, 

« 



* 

* 

but also to bo able to draw upon other disciplines encountered through . 
general education and minor field concentrations. *• For example, If 
biology majors are Investigating whether a bridge should be'constructed -**~ 
across an estuary, 'thpy should be able to determine the potential 'Impact 
such a structure could have on the flora and fauna of the local ecosystem 
as wel\,as to draw from the Humanities and Social Sciences to address 
possible economic,' social, and aesthetic Implications. 

The cue function of an assessment task at the third level also' 
Involves Imposing order arid more structure on more ambiguous situations 
that, at the proximal or second level. A wider repertoire of cognitive ' 
skills 1s also required to perform tasks at this level and responses entail 
the Integration of a wider diversity of principles and supportive detail. 
Students must also be aware of the biases and. values of the respective 
disciplines as well as be congnlzarit of their own personal values. They 
should be able to cope with value conflicts by applying a higher order of 
recognized sets of principles and ethics. 

The fourth or . ultimate level generic skill assessment concerns eva- '• 
luatlng whether students are able to "break 1 ! the boundaries of disciplinary 

• 

thought as acquired 1n sequences of academic courses and are able to 
demonstrate the ability to use generic skills to cope with real life Issues 
and problems such as choosing a political candidate, selecting a career, 
resolving a personal conflict, making decisions 1n professional 
situations, or taking a stand on a political Issue. .In such assessments, 
students are presented with a novel situation and are required to structure 
its apparent elements, seek appropriate Information, formulate rules when 
necessary, and to distinguish among personal, cultural, and disciplinary 



; ■• ' '• ■-' . 4l2 

n • -j 

values.. \ At this level, the assessment tasks 



may well appear to be 



"discipline-free" in the sense th^4rresponses^dO: not require the recalling 

facts, definitions, concepts* or principles -from Ja single academic • 

discipline^ Students are required to perform a more difficult task —to 

seek out, select and 'evaluate informatioji pertaining to an issue, a 

./ ■ . 

problem, or pending decision. / 

At. this ultimate level of "scope" dimension ,*the stimuli of the' 

assessment tasks tend to be. ambiguous and unstructured. Students are asked 

to assume a socially responsible role presented with an issue, a dilemma, a 

question, a. problem to solve or a decision to ; make. The task should "be 

sufficiently complex so as to require "the integration bf a variety of 

disciplinary perspectives an^J skills. Students from v the respective y 

disciplines are likely to demonstrate differences in the mahn'er in^ which ^ 

they respond to prot)lem-sq,lvihgi situations because the perspectives and - ^ 

analytical '"triors' 1 . (Commuhications, Disciplined Inquiry/ Valuing and 

Execution) they have acquired will reflect the kinds of educational 

experiences they have had previously.- All three exercise^ in the prototype 

test assess generic skills at the ultimate level, each drawing upon 

perspectives from a variety of disciplines; 1 




Assumptions 

Three assumptions or propostionsj are ma^ev in the development of the 

prototype baccalaureate outcome measure: * 

V there is a continuum of cognitive cop lex ity that "underlies each 
generic skill dimension which is. reflected in terms' of high, 1 
medium ^ndrlow levels of performance; ' r 



* An aim of higher education i's to assist student s'^fn developing 
problem solving capabilities which, necessarily includes important 



/ component skills (i*e. generic skills) ; .-; v v /r**. 

•• - - • ■>..,.;..,.- ■ .y- '.''■*..•.••■;■■. - : f* ■■ - " ■ 

, * One criterion with which to evaluate, academic program*effec- 

^ r ■ ■ tiyfeness should include an assessment of ^growth in generic skil 1 ' .'• 

*•'•■■ capabilities of students. ^ v/,';; - ''• ;; - v " 

' ' ■• Method . . . 

Jest Development . . - * : ' v 



* The scenarios . Three scenarios were developed using the Ski 1 1 X 
Content X Scope model depicted in Figure 3. The first exercise, Cuban 
Crisis, was designed toemphasize Humanities and Social Science perspec- '-/_'. 
tiyes of the cube; the second, Land Development, was designed to draw on 
the Natural and Social Sciences; and "third, Token Economy draws on Social 
^Sciences and Professional education. Cuban Crisis was developed with the 
assistance of a faculty col league in the Department of Religion^ Land ' 
Development was developed by the author from a series of articles and edi- 
torials in a local newspaper; and Token Economy was also formulated by the 
author. Each of the scenarios requires an individual to assume a social 
role such as a social caseworker (Cuban Crisis) , a county corranissioner 
(Land Development), and a county school board member (Token Economy) . The 
scenarios were intended to create interesting and engaging situations to' 
help sustain motivation in. the task. The situations in the scenarios were : 
designed to be sufficiently complex so as to compel high levels of "per for- 
mance. for of each of the generic skills. The three scenarios contained 
between 150 and 250 words and are located in Appendix III.' - 

The directive statements . The directive statements are the keys to 
the effectiveness of each task in terms of eliciting maximum performance in 
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/ the respective generic skills-, the directive^v^re derived from libera] 

• arts competency statements obtained from ear llej^e^eirch efforts (Peterson 

and Watkins 1978 and Peterson, 1982). They are the same for each scenario 

and are as follows: . - . - v - ' 

. a r 1 . ." Problem Solving • » • ' / . 

Subskill: Formulate a plan of action to rectify a nfeed o^anomaly 
■ ■■ based on. a higher order, principled rationale. 

> Directive 'Statement : Describe 6/hich course(s) of action you .would 

' * recommend in the aboye situation and explain 
why. Please elaborate as much as you can*- 
1 and use the back of the paper if necessary. 

2. Communication . — \ v 




Subskill? Comprehend/complex situations by expressing the main 
idea in one 's own words. 

Directive: As a, ( role specified) what is the central issue for 
you in the above scenario? -^N 



3. Analyses 
Subskill: 



Perceive elements 4 in a problem situation by demon- 
strating tha ability to view an event from multiple 
perspectives. 



Directive: Describe the problem from A's perspective^ B/s perV 
spective or C's perspective. 

4. Synthesis 

Subskill: Formula'te-a variety of discretely plausible courses 
_r of action given a problem situation. 



Directive: Li^t as many possible^courses of action that might be 
takfen in the above scenario, including options you 
would not choose to follow. Use the back of the 
page if there is not enough space -for a VI of your 
alternatives. 



5. Valuing 

Subskill: Infer values from behavior 
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. "■ .. •. ■ . . " .15 

* Directive What values 'underlie A's behavior, B's behavior, and 

. C's behavior? : : . 



6. Execution 



Subskitl : Perceive logical mfcans-ends relationships with suf- , 
^ ficient detail. 

Directive: For the solution you recommended in Part I (Problem ,"• 
Solving) , ouMine the sequence of actions that you 
... ; would take to implement the solution. Describe what" 

II ■ ! you would do "first, second, third, etc. 

the rating scales . "fRe development of rating scales involved the 

delineation of performance attributes for high (5) , medium (3) and low (1) 

anchor points' along a 5 -point continuum. The works of developmental 

psychologists such as Piaget (See Havel, 1963), Kohjberg (1971) and Perry 

(1970) were helpful in developing a 4 priori general conceptualizations of 

the respective performance continiia. The --d^ 

developed andVefined on the basis of a pilot test. The scenario and the 
directives for Cuban Crises (Exercise 1) are presented in Appendix I. The 
rating scales appear in Appendix, II. 

Test battery assembly . The prototype baccalaureate test consists of a 
background information form, a vocabulary test, and three generic skills 
exercises. A.5-point rating scale was appended to Cuban Crisis and Land 
Development to assess the degree of f ami 1 i ar i ty With the content of the/;;;'/;V 
respective scenarios. /The background information form asked the sub- 
jects 1 age, total number of semester hours cpmp leted , the number of 
semester hours completed in their .major field and the ; n umbtgry ;of seme s ter . >; 
hours completed in" Psychology . Tffe vocabulary test was the; Wide Range 
Vocabulary Test (1962) , a 12-minute test to gain an indication of a -general 
intelligence factor (Spearman g) that ostensibly underlies the demonstra- . ; 



• • • it. . . 
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tlon of- generic skilTs according to the model, This short vocabulary test 
pan be used in program "evaluation purpdses as*a covari ate to equate perfor- 
roances among cohort groups in terms of verbal ability. Each exercise con- 
sists of two parts: (1) the decision (Problem Solving) and (2) -the 
fnquiry (the assessment of generic skills) . The 'first part 9 the decision, 
seeks to establish a goal and rationale for coping with a problem situation 
while the second part, the inquiry, sqeks to ''assess the level of generic 
skill proficiency undergirding the decision. The procedure is somewhat 
analogous to the procedures used * to administer psychological projective 

Instruments . , .. y v 

Pilot testing . . The pilot test subjects consisted of 17 juniors and 
r senior from an upper divisfon Human Relations and Communication 
offered in the College of Education and 5 graduate students in Counseling. 
Both groups completed the three exercises as homework assignments. As a 
result of the pilot test, both the directive statements and- mbdel responses 
at the 5, 3 and 1 anchor points were revised. - . v 

Field-Test Administration 

4 Samples . Three groups of students were sought for the field test: 
lower division, upper .division and graduate students to g3fin a full range 
of generic skill performances and background characteristics with-which to 
analyze relationships among constructs. The lower-division students were 
volunteers from Introductory Psychology classes who received course credit 
"for participation 'in faculty research. The upper division students were 
.paid volunteers ($10 per session) who came primarily from ROTC classes and 



" Human Relations and Communication classes. 'Students from almost jal 1 majors 

• .attend these classes. The graduate students were volunteers from the 
author s's graduate classes in the 'College of Education, most of whom- were 
in their first year of graduate school . The sample consisted of 20 fresh- 
men and sophomores, 26 juniors arid seniors', and- 16 graduate students. 
There were 27 males and 35 females. Major fit Id representation was 61% 

- Professional Schools,, 25% Social Sciences,, 5% Natural Sciences and 5% 
Undecided. The author recognizes that a stratified random sampling proce- 
'dure would have been preferred to offset the risk of sampling error bias. 

' Testing sessions . When visiting classes to secure volunteers, a sign- 
up sheet was circulated for subjects to attend scheduled testing sessions. 
Subjects" were informed that a test i ng sess i on ' wou ld 4 take between 2 and 3 
hours. The show-rate for Psychology students and graduate students (non 
paid valunteers) was 75% while only 50% of the upper division students 
(paid volunteers) appeared for testing after signing up. 

When subjects arrived for the testing sessions, they first completed : 
the background information questionnaire and then the 12-minute Wide Range 
Vocabulary Test . They were then informed they had unlimited time to 
complete the three generic skills exercises. A brief description of a 
token econbrny was given oraTly by the proctor since some of the subjects 
were unfamiliar with what a token economy was. Mbst students completed the 
exercises with in two hours and no student took longer than three . The v 
order in which the exercises were administered was Cuban Crisis, Land 
Development and Token Economy. 
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Rating the Responses 

Each response was rated three times by three ABD doctoral students in 
Counseling. Before beginning the rating procedures, they were informed 
about the nature of the project, the skills and the scales. They were \ 
instructed, for each -of the dimensions, to first rate the/ responses of the 
first ten subjects and then'to return to the beginning and readjust any 
scores following a second reading. Then, following a question and answer 
period, the remaining 52 responses were scored. Approximately seven hours 
were required of each rater to score 62 samples for all six generic skill 
dimensions for each exercise. Thus about 21 hours of rating time was 
required for each ' rater to evaluate a 1*1 three exercises for 62 examinees. . . 
The raters evaluated all protocols one dimension at a time to eliminate „ 
halo effects across skills. The tests were sequenced in alphabetical order 
and the identity of subjects remained anonymous to the raters. To offset 
potential ordering effects, one rater scored a given set responses forward, 
another backward and a third began in the middle and proceeded forward. 

. Results' , 

Test difficulty ./ The means and standard deviations and range for each 
exercise as well as, the scores for all three exercises combined are pre- • , 
sented in Table 1. All of the means were near the midrange of the scales 
(I.e. medium level responses) except for the Valuing dimension which was 
between the medium and low range (2 level responses) . ATI 6f the distribu- 
tions met normality criteria for skew and kurtosis. The complete range of 
performances (lowest possible (3) and highest possible (15)) were found in 
twelve of eighteen tasks (three exercises X six skills). The ranges of 
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Table 1 



Difficulty level, range and reliability 



Skills 



Exercises 



Prob tern Solving 
. 'Exercise 1 
Exejrcise 2 
Exercise 3 

•Combined 

CjPCTiuri' ljc ation 
^Txirclse* i 
• Exercise 2 

Exercise 3 

Combined 



Analysis 



Exi 



xercise 
Exercise 
Exerc i se 
Combined 



1 

2 
3 



Mean* 



S.D. 



Maximum 

Possible Observed 
Scale Range 



< Mean 
Alpha- Interrater 
Reliability Reliability 



8.18 


2.85 


15 


:--V 3-15 


-79 


.56 


8.15 


2.78 


15 


V 3-15 


.78 


. .54 


8.08_ 


2.87 


15 


3-15 


.78 


.55 


24.40 

*. i 


6.44 . 


45 




* - 8 V' 




7.52 


2.93 


^15/ 


3-13 


.79 


.57 


9.45 


3.52 


15 / 


3-15 


V .86 


.67 


7.90^ 


2.94 


15 


3-15 


.75 


.50 


24.87 


7.41 . 


45/ 


10-43 

• . . v& 


.84 




9.00 


2.57 


15V 


4-14 


.63 '. 


.35 


8.73- K60 


15 


3-13 •- 


.67 


.46 


8.10 


2.82 


. 15 . 


3-15 


.73 • 


.48 


24.84, 


5.20 


45 


15-40' 


1 .73 





Synthesis 

Exercise 1 7.34 2.33 .15 

Exerciser 9.65 ' 2.82 15 

Exercise 3 8.73 3.09 15 

Combined 25.71 6.17 45 



3-13 
' 3-lv5 

3-15^ 
15-41 



.67 
.82 
.80 
.80 



.41 
.60 
.56 



V aluing 

Exerc i se 1 
Exercise 2. 
Exercise 3' 
Combined 

Execution 
Exercise 1 
Exercise 2 
\ Exercise 3 
': XCombined 



5.81. 


2.81 


15 


3-15 


.82 


.60 


6.56 


2.34 


15 


3-13 


.69 


.44 


6.32 


3.03 


15 


3-15 


-.88 


— .71 


18.69 


6.47 

• 


45 


9-33 


.85 • 




7.00 


2 .41 * 


15 


' 3-13 


.66 


.41 


8.42 


2.64 


15 


3-15 


.68 


.43 


8.74 


3.27 


15 


3-15 ; 


.89 




24.16 


6.12 


"45 


12-39 


.78 ' ! '" 





\ Exercise 
! Exercise 2 
I Exercise 3 



1 = 



Cuban Crisis 
Land Development 
Token Economy 



er|c 
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scores of the combined scales created by summing nine ratings across three 

' ■ . • * •»-..■» 

a ■ 

exercises were from 24 to 33. In only one 'skill, Valuing, did a student 
receive nine, t-ratings across all three exercises. No student earned all 
S's across all the exercises for any of the skills. One student in 
Analysis earned seven,? 5 ratings and two, 4 ratings. Therefore, the levels, 

of difficulty, ranges of scores and variability of scores within generic 

« ! ° . ■ * - . ... ' 

skill dimensions appear to be sufficient for use as a program evaluation 

measure. . ' , : ■ \ ' ■ 

Test reliability . The reliabilities (coefficient alpha), for the scale 
of 18 tasks (3 exercises x 6 skills) are also presented in Table 1. The 
alphas 4 range from .63 to .89 with a mean of .71. The alphas for the com- 
bined scales formed by adding nine ratings for each subject ranged from .73 
? to .85 with a mean of .80. The interrater product ^moiren^ 
each of the 18 individual tasks ranged from .35 to .73 Wjith a mean of .53. 
This le>>e1 of interrat^ agreement is typical of measures using high levels 
of inference to rate the responses. Th6 alpha reliabilities of the com- 
bined ratings were viewed as sufficient for use of the instrument in 
program evaluation. 

The relationships between exercises . Both zero-order and attenuated 
correlations of skills across exercises are shown in Table 2 on the next ' 
page. The range for zero-order correlations is .19 to .49 with a mean of 
.37. Likewise the range for the attenuated correlations is .26 tp .65 with 
a mean of .49. These coefficients represent an index of the degree of com- 
monality of ski lis, across exercises. Using the attenuated coefficients to 
estimate the "relationship between two variables assuming perfect measures, 
the average amount of common variance between two identical skills across 



Zero -Order and Attenuated Correlations by Skills Between Exercises 



Skills- 

Exercises (r£t) (r£t) 



r xy r xy 



. Skills 



r xy, 



Exercises (rtt) . (rtt) 



Problem Solving 


' ? Ex2 


Ex3 


Exercise (1)" 


Sr. 

.31 .44 
\ (.40) a (.57> 


j 

w Exercise (2) 




.32 
(.41) 


Communication 


Ex2 


Ex3 


Exercise (1) 


.49 
(.60) 


.49. 
(.65) 


Exercise (2) 




.33 
(.41) 


Analysis 


Ex2 


Ex3 


Exercise (1) 


.37 
* (.57) 


.33 
(.49) 


Exercise (2) 




.24 
(.34) 



Synthesis * 
Exercise (1) 

Exercise (2) 

* ■■ . 
Valuing 

* Exercies (1) 
Exercise (2) 

Execution 

Exercise [1) 

i 

Exercise (2) 



Ex2 
.19 



Ex2 



Ex2 

.37 
(.55) 



Ex3 



.36 

* ( .26) (.49) 

.43 
( .58) 



Ex3 



.49 .36 
(.65) (.42) 

.48 
(.62) 



Ex3 

.37 « 
( .48) 

.20 
(.26) 



0 



a. Numbers in parentheses are correlations for attenuation based on alpha 
reliabilities. " * 



different exercises is 24%. With zero-order correlations; the average 
amount of common variance between skills decreases to 14%. Further, even 
though the exercies were designed to represent different content domains, 

oh. the skill x content x. scope matrix, the range of strength of attenuated 

\ > ••■*■*.■. - 

intercorrelations within skill areas appears small (maximum .22 between 

-highest and lowest for Synthesis). Therefore, because of the modest 

•I. * * •- 

correlations (i".e. shared variances) across exercises within skills, enti- 

ties called generic skills were measured. *• 

The relationship among generic skills . The matrix .,f or both zero-order 
and attenuated correlations among generic skills is presented in Table 3. 
The generic skill scores were determined by summing the nine ratings across 
three exercises (three ratings for each of three exerdises). The range of 
zero-order correlations among skills is .33 to .66 with a mean of .55 while 
the ranj^ for attenuated correlations is .41 to .84 with a mean correlation 
of .69. -Therefore, using the mean attenuated correlation among all generic 
skills, there is an averagS of 48% common variance between individual' 4 
skills. While this may seem high at first glance, one must be mindful that 
there is also .an. average of 52% unique variance between the skills, thus,, 
the six individual skill dimensions may be considered as independent and 
meaningful constructs. 

Familiarity with problem situation . An investigation was made 
regarding the degree to which familiarity with the problem scenario in 
real^life situations (Cubah Crisis and Land Development) or the extent to 
which of coursework in a technical situation (Token Economy) might be 
related to generic skill performance. In the two Real-Life exercises, 



23 



Table 3 - 
Zero Order and Attenuated Correlations' Among Generic Skills 



Exec* 
r^y 

(nt) - 



Com 
• r^y 

Generic Skills (rtt) 



Anal 



Syn v 



Problem Solving 

Communication 

Analyses 

Synthesis 

Valuing 



.61 
(.74)3 



.63 
(.82) 

.52 
(.66) 



.63 
( .78) 

.50 
( .61) 

.60 
(.79) 



.60 • 
(.72) 

.45 
(.53) 

.66 
(.84) 

.63 
(.77) 



.49 
(.62) 

.33 
(.41) 

.63* 
(.84) 

.48 
(.61) 

.49 
(.60) 



3K Numbers in parentheses are correlations corrected for attenuation based on 
alpha reliabilities. 
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Cuban Crisis and Land Development, subjects rated their level of f ami- 
liarity with the two situations on a five point scale (see Appendix II). 
Jn the third exercise, Token Economy, subjects were asked to record'the 
total number of semester hours completed in Psychology. The correlations 
between responses to the familiarity scales and the first two exercises as 

well as credit hours taken in Psychology with respect to the third exercise 

t « • 

are presented in Table 4. The results indicate there was only one exercise* 
within one skill in which there was a significant correlation between the 
familiarity, scale and generic skill performance. This one significant 
correlation ^was considered to be an artifact of the data- Therefore it was 
concluded tha^ familiarity with an event in real -life scenarios in this 

battery has little if no bearing on the demonstration of generic skills. 

\ • ' 

Significant correlations were however obtained between credit hours in 

\ • 

psychology and generic skill scores in. the Token Economy Exercise. When 
W^erpreting these results, it is well to refer to Table 5 on page 26 in 
which correlations between generic skills and a variety of background fac- 
tors are presented. • The strength of correlations .between generic skills 
and age and total credit hours completed appears strong. Thus the rela- 
tionship between credit hours taken in Psychology and generic skill scores 
may be more owed to maturation, total educational experience and verbal 
• ability than to mastery of content ir\ psychology. ' On the basis of these 
data the conclusion cannot be made in terms of whether the mastery of spe- 
cific content is related to gerleric skill performance. Thus the rela- 
tionship between the degree of content mastery and generic skills is still 
an open Issue. Jhe implications of this finding for academic program eva- 
luation are discussed later. % \ 
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Table 4 



Correlations of Familiarity Wi.th Problem Situation or Content Domain )* 



Skills 



Exercise 



Skills 



~ F 

Exercise 



Problem Solving 
, Exercise 1 
Exercise 2 
Exercise 3 



.03a 

'• 06 L* 
.49 c 



Synthesis 
Exercise 1 
Exercise 2 
Exercise 3 



.01 
.08 
.31** 



Communication 
Exercise 1 
Exercise 2 
Exercise 3 



.06 
.06 
.28* 



Valuing 
Exercise 
Exercise 
Exercis; 



.06 " 
.10 

.38*** 



Analysis 



Exercise 1 
Exercise 2 
Exercise 3 



.22* 
.09 

.52*** 



Execution 
Exercise 
Exercise 
Exercise 



.00 
.03 

,45*** 



a) Correlation with 5 point familiarity scale with Cuban Culture 

b) Correlation with 5 point familiarity scale with Land Development 
Issues • • i . 

c) Correlation with total semester hours completed in Psychology 

* p < .05 

** p < .01 I' ' 

*** p < .001 ' 
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Relationship between generic skills and verbal ability, age and 
educational' experience . Correlations between age, elements of educational 
experience and vocabulary are presented in Table 5. The mean correlations 
between- generic skill performance and age is .56, year in college .52* 
total semester hours >. 52, semester hours in major .49; semester hours in 
Psychology .49, semester hours of general education .34 and vocabulary .50. 
Apparently, educational experience, maturation and intellectual factors- are 
all related td^neric skill performance. Unfortunately all of these fac- 
tors are themselves highly interrelated in so that the determination of 
their relative 'contribution to generic sk i 11 performance could not be ' 
ascertained with regression .analyses due to the presence of mu It ico 1 i n- 
earity (Lewis-Beck, 1980) . If the subject pool had included a group of 
adult learners with limited educational experience, then perhaps some light 
could have been shed on the nature-nurture controversy with inspect to the ' 
demonstration of generic skills. Nevertheless these' results point out that- 
the use of a general intelligence measure should:' be included as a covariate 
when comparing generic skill performance between or among, student cohorts 
in academic program evaluation. 

Discussion 

A prototype baccalaureate outcome measure was developed to make the ' 
assessment of higher-order thinking skills more theoretically coherant, 
^practical, andpsychometrically defensible. A problem solving model was ^ 
advanced in which five generic skills were identified which undergird the 
capstone skill, Problem Solving, namely Communication Analysis, Synthesis, 
Valuing and Execution. An assessment strategy was advanced in which .all of 
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Solving Com. Anal. Syn. Val. Exec. 



Age 


.65 


.'37 


.60 


.61 


.56 


.51 


Year in College 


.54 


.31 


.58, 


.58 


.56 


.'53 


Total Semester 
Hours Completed 


.56 


.35 


.57 


^60 


.57 


.49 


Semester Hours 
1n Major 


.54 


.33 


.55 


.56 


.53 


.46 


Semester Hours 
- 1n Psychology 


.53 - 


.27 


.60 


.43 


,49 


. .60 


General Education 


.34 


.22 


.34 


.39 


.38 


.34 


Vocabulary 


.45 


.31 


.63 


.55 


.62 


•46 
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these skills may be treated in the context of one problem solving 
situation. The following discussion focuses on the validity, reliability 
and utility use of the measure in academic program evaluation. 

Validity as a generic skill measure . At the present stage in the 
development of generic skills .measures, a major concern regarding vali- 
dity is from the standpoint of construct validity. Data were analyzed With 
respect to the independence of dimensions, commonality of skills across 
tasks, the contribution of intelligence factors, and\the potential bias \ 
stimulus familiarity. It would ^appear from the results of analysis that 
the constructs advanced by the model possess an adequate degree of indepen- 
dence. The testing of th^constructs are not limited to the specific 
directives used in the^present prototype instrument. Certainly other 
directives could be used in the scenarios to assess different component 
skills. In fact, other component skills were contemplated in the develop- 
ment of the persent instrument. These happened Jto be some of the most fre- 
quently identified component skills from previous studies (Peterson, 1982, 
Peterson and Watkins, 1978). In this regard, the author is not satisfied 
with the Valuing directive. The directive used in this battery represents, 
a skill too far down 'in a hierarchial analysis of the Valuing dimension. 
Nevertheless this particular directive pertaining to the infering of values 
from behavior is almost universal in the lists of generic skills proposed 
by liberal arts faculties. ■*> 

These generic, ski lis "do cut across assessment tasks. A recent study 
by Sackett and Dreher (1982) found that the mean correlation among typical'- 
assessment- center tasks (In basket, Leaderless group, etc.) was near zero, 



raising questions concerning whether the concept of generic skills is rele- 
yant. However, the variability in the kinds of exercises in the present 

prototype measure is much less than in the typical "bill-of-falre" in 

... .. <? . ■ 

assessment center tasks. Nevertheless* an issue is raised for further 
research, regafcHng lirtiits of generic skills in terms of decreasing task 
fidelity or epistomologicgl distance. Had one of the tasks^gauired mathe- 
matical proficiency, would the intercorrelations among exercises be 
reduced? The fact that the average attenuated correlation different among 
skills (rtt = -69) was greater than the average attenuated correlation 
between the same skills across tasks (rtt=.49) gives support for further 
inquiry into the meaning of the concept generic . . 

Another interesting finding was the extent" to which the skills co-vary 
with educational experie nce and age. Cattell (1977) an d Horn (1981) pro - 
pose that general ability consists of two abilities: Fluid and Crystallized 
^abi-U-Ues-^^ f igural analogies, classi- 



fications, series completion problems, memory span ajpd the like^ begin 
their development early in life and reach a peak in late adolescence and 
then begin to decline in early to middle adulthood. Crystallized abilities 
on the other hand, often measured by tests of vocabulary, reading compre- 
hens ion, semantic relations; and general information, tend to Increase inde- 
finitely with agev These latter abilities appear to be similar to the ways 
in which generic skills, as measured in this battery > perform. It seems-; 
logical that educational experience would have : an impact on the development 
of crystal ized abi lilies but not on . f 1 u i d ab i 1 i t i e s • j i IfeV ier t h e 1 ess, r thie ^ ; " 
relationship between generic skills and fluid abilities is point of 
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Inquiry. The use of the Wide Range Vocabulary test was an attempt to 
accountfor a fundamental attribute (i.e. a fluid ability) of higRer order 
intellectual performance but it may actually be another construct in an 
array of crystallized abilities as suggested by Willis, Schail and. Lueers 
(1983). Nevertheless, when using the battery to compare the "development 
generic skills across academic disciplines or between different classes, ; 
some measure should seemingly be used to equate potential differences in 
academic aptitudes. The Wide Range Vocabulary Test would ostensibly be an 

s Indicator of verbal aptitude. / ', * 

Irresistably, one last "as if" analysis was conducted to determine 
whether there'would be significant differences between lower division, 
upper division and graduate students in this sample population in terms of 

-generic skill performance. In no "way should these results be construed as: 
representing 'these respective groups at Florida State University. The sub- 
jects were not drawn at random from circumscribed populations. Neverthe- 
less, data are presented to demonstrate how the measure might be used to 
evaluate the growth of generic problem-solving skills during the 
baccalaureate years. Verbal ability as measured bv the ETS Wide Range 1 
Vocabulary Test was used as a covarlate to equate verbal afiility amoncj •• c 
these educational levels. r 

The results of this "demonstration" analysis are portrayed in Table 6 
on the next page. One can note there' were no differences between lower and 
upper division performances but that graduate student performances across 
five or six skills were significantly different from undergrjacJuates.. Could 
it be that undergraduate education enhances the development of cognitive 
schemas on which to structure and interpret subsequent 1/ife experience. 
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Table 6 

Generic Skills by Performance by Educational Level 
with Vocabulary Score as Covari ate. 



* 

Skill 


Lower 
Division 

-CN ■■ 20) 


Upper 
Division 
(N =26) 


• Graduate 
fN =16) ' 


F 


froble Solving 


M 

SD 


22. 6 a 
5.0 


21.8 
4.8 


. 30.9 
6.1 


. mm ^% mm . J . t _ i _ 

7 • 97 *** ■ 


Conmunication 


M 

SD 


23.8 
5.2 


23.2 . 

8 - 6 C 


28.9 
6.4 


.88 


/Analysis 


M 

SD 


24.1 ^ 
3.1 


23.9 
< 4.7 


31.3 
4.4 


4.74* 


Synthesis 


M 

SD 


23.5 
3.4 


23.5 
5.3 


32.1 
5.9 


6.18** 


Valuing 


M 

SD 


15.6 
3.3 


17.3 
5.1 

* 


24.9 
7.4 


4.80* 


Execution 


M 

SD 


22.0 
-4.5 


21.9 
, 4.5 


30.5 
" 5.6 


8.39*** 



a. Unadjusted Means 



* p < .05 
** p < .01 
*** p f .001 
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Perhaps a requisite amount of life's experiences is required before higher- 
order thought processes become evident in the real-life level problem 
situations. It coQld be that problem ^scenarios cast at the major field 
level would discriminate between lower and and upper division students. 

Reliability . The battery appear r. to possess sufficient reliability 
for use as a program evaluation instrument. The ^modest interrater product- 
moment correlations could be improved by a more concentrated/effort to 
t^ain raters and to provide an honorarium to foster motivation to mak|| 
subtle discriminations among performances. With such procedures,, the/ 
assessment battery could be reduced to two exercises and rated by two 
raters without appreciably lowering reliability. The bost of admlnis- 
tering and scoring would also be reduced. 

Utility . Assuming that the instrument is sufficiently valid and 
reliable for use in academic program evaluation, administrators Ultimately 
inquire about the costs to the institution, the faculty and to students. 
The estimated direct costs to develop, administer, score and analyze; the 
results of this instrument for a subject pool of 62 was $6000. This amount 
does not include research costs involved prior to the actual development of 
the measure. It must be remembered however that costs escalate with the 
desire to achieve greater reliability in scores And greater confi^ce that 
a sample respresents a given population. In order to obtain samples of 

freshmen and seniors that possess a sampling errdr of less than .05 and a 

'">•... t *,'. : ' ' ■ .w 

confideate level of .10,. as many as 400 subjects would be required (Cochran 

1903k At this confidence level, th£ cost of evaluation escalates to 

$15,000 assuming ;3 exercises, 3 raters, and paying each subject $10 to ; 

achieve a desired show-rate 4 f or stratified random sampling procedures. . 



With two exercises and two ratings for each exercise the cost 1s $11,000. 

i ' • . ■ ■ • 

The major variable costs, of course, are scoring and paying students to 

take the test to achieves desired sampling accuracy. Costs of course are 

reduced as these two factors are compromised. Costs for instrument deve-; 

* 7 • • ■ . . . . • ■ . 

lopment, computer analysis, clerical support and supplies are nearly fixe3 v 
and /these were estimated to be $4500. - 

j Conclusion . Without a doubt, performance testing is expensive com- 
pared to using commercially prepared multiple choice tests. The usefulness 
of performance tests will depend on whether the kinds of information about 
stiidents are useful for making curriculum decisions pertaining to course 
structure and student degree requirements. A previous study (Peterson, 
1982) reported that when performance tests and multiple choice tests are 
combined in an assessment battery, these two kinds of assessments load on 
two distinct factors suggesting thaf performance tests do provide different 
Information from multiple choice tests. A prototype baccalaureate measure 
has been developed thfct appears to be associated with the level of educa- 
tional attainment 7 (i.e. total credifhours completed and level of 

-educational attainment). However, the relative contributions of matura- 
tion, certain intellectual factors and various educational experiences to 
generic skill performance should be ascertained through experimental 
research paradigms before curriculum decisions should be based on its use. 
Nevertheless, this prototype measure appears to be worthy of further * • : 
exploration as^a research instrument with^D^ 
the relationship between curriculum and outcomes of post secondary 

•education.- " ■■:•/■•■•. , • . 
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APPENDIX I 

Real Life Problem (Solving Exercise 
"Cuban Crisis" 

* PART I: The Decision, 

The Scenario 

As a case worker for the Department of Social Services, you have been 
assigned- to work with a young Cuban woman, found beaten and semi-conscious 
1 n a Cuban neighborhood. Taken by police to a hospital, she acknowledged 
that she had been sexually assaulted, but refused to Identify her attacker; 
she became hysterical at the suggestion that she undergo a physical exami- 
nation to determine 1f rape had occurred. Her family has refused to 
cooperate with police attempts to Investigate the presumed rape, although 
the police have been able to determine her attacker's Identity by Infor- 
mants 1n the Cuban community. The girl's father has flatly refused to 
swear out a warrant, and the girl refused to acknowledge either that a rape 
has occurred or her assailant's Identity. Frustrated, the police have 
called you to come to the police station and take over the case. . 

Directions : . , 

Your task" 1s to describe which course(s) of action you would recommend 
be taken and why . Please elaborate, as much as you can. Use the back of 
this page if necessary. 



Part II: The Inquiry ( 

The Scenario , ' 

— ■ < 

As a case worker for the Department of Social Services, you have been 
assigned to work with a young Cuban woman, found beaten and semi -conscious 
1n a Cuban, neighborhood 1n Miami, Fla. Taken by police to -a hospital, she 
acknowledged that she had been sexually assaulted, but refused to Identify 
,her attacker; she became hysterical at the suggestion that she undergo a . , 
physical examination to determine 1f rape had occurred. Her family has 
refused to cooperate\w1th police attempts to Investigate the presumed.. rape, 
although the police have been able to determine her attack'er's Identity by 
Informants 1n the Cuban community. The girl's father has flatly refused to 
swear out a warrant, and the girl refused to acknowledge either that a rape 
has occurred or her assailant's Identity. Frustrated, the police have 
called you to come to the police station and take over the case. 

Directions : 

Please answer the following questions as completely and concisely as 
you can. However, 1f there 1s not enough space, please ask the proctor for 
additional paper. 

Communication J . • •, M 

T. — As a social worker, what are the central Issues for you 1n the above 

scenario? < 



i 2" aly De scribe the problem from the girl's perspective, from the father's 
perspective, from the police's perspective. • 



(a) The girl's: 



(b) The father's: 



(c) The police: 
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Synthe sis * t 

S4, List as many possible courses of action that might be taken in the 

above scenario, IncludlngfdpTTons you would not choose to follow. Use 
the back of this page 1f /here 1s not, enought space for all of your 
alternatives. 

• 'L*l 

... is ; : 



Valuing 

4. What values\m1ght you hypothesize underlie the father's behavior?. The 



s\ mi g 

girl's behavloif? 

\ » 

Father's values:- 



Girl's values; >'■ 



S : 

Execution, • < ' ' 

5u Forthe solution you recommended on page 1, outline the, sequence of 

actions that you would takie to Implement the solution. Describe what 

you would do first, second, third; etc. (Go on to the back of this 
page for additional steps). 



ill 



(3) 




f 

• • 


II <£ _ .. 



SI 



1 
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Familiarity Scale 



Indicate your familiarity with Latin or Cuban culture by circling the . 
number which most closely approximates your acquaintance. 

1. know nothing (e.g., you have no knowledge of Cuban or Latin 

culture nor how these cultures are different front 
Middle America, ■ , 

2. only a vag\ie familiarity (e.g., your familiarity 1s restricted to 

i what you have read 1n the newspapers or have seen 
on television news broadcasts) 

3. somewhat familiar (e.g., you have studied the culture but do not 

speak or .read Spanish) 

4. familiar (e.g., you have studied. the culture and have taken 

courses Including Spanish) * 

5. very familiar (e.gii, you have first hand acquaintance through 

living 1n the culture for 6 months or more and 
speak the language. You are very familiar. with 
ways 1n which the culture 1s different from 
Middle America.) 



°. •. - " : :; Generic Skills Rating Scales. 
. , ' 5 • : /: Cuban Crisis- . 

K Generalized Problem Solving (Formulate a plan of action to rectifya^nfeed or anomaly 
. ' based on a higher order, principletrntionale): ; £^ 

High • .'■ ftedium : - 

--—5 ,. . " .. y—* ^ 

Options selected. and reasons Solution attempts to satisfy : Solution is vague, ambiguous,' 

given are* logical and demon- everyone 1 s rieeds such .that no trial and error strategies 

strate capacity for higher . 'one's needs are completely met*, used, Solution does not 

order principled thought. ■> Or, girl 's psychological well t ak e i n to s acc ou n t mu 1 1 i p 1 e 

Th% major concern^to others . being sacrificed for appre- perspectives' and.:value's 

should be the girl's, psycho- ■ hension« of rapist; .However, ) ; operating/ in the 'drama, ;*p 

logical well-being.- The ; their must be some concern for Favors one. view with, out : , . 

proposed action. is concrete girl 's psychological well- . ; justification. Solution de- ' 

(as opposed to vague, abstract) being. At this level , there is monstrates- no. regard for' 

and several perspectives are nd indication of a balance ' girl's well-being, eg. "Get 

broiigh to bear on the decision ' among perspective, . Focus is the rapist", 
There is a consideration of only on one part of scenario, 
social 'and cultural factors . \ i .& the girl 's welfare or - 

bringing the assailant •• 

trial. ■■ ^A : t*^^M 

II . Conmunication ( Understands of complex situations by expressing the main idea In ones own words.) 

High ' . . Medium \ • ' Low V 

. .. ; -5 ^ . : 4 . ,; ■ :};-^y'^'^ 

Mentions that the first First emphasis is nn capturing Responds to manifest tissues, 

.V priority is the girl's ; and punishing rapist. Mentions eg, "getting a.11 parties to 

psychological well being. * concern for girl's well being, < cooperate wilh police",: No 
Answer reflects sensitivity ■ ] cultural issues,; father! 

to Cuban cultural issues ' for support, but these are . ; No awareness, "flf cultural , : 

( "machismo",' "chipercning", secondary, Answer reflects .:• issues, intolerant, 'preju- 

etc), Mentions father's need incomplete or superficial diced biased; Extremely 

to support and answer addresses awareness of underlying issues . superficial understanding of ' 

. society's responsibilities to eg, "Help the girl and family the complexities of the-; :. ; f 

individuals in need. ,-. feel better", - ?fj -i situation. Conple^e^ly^: " 

Sensitivity to "deeper", i - ; : : : \ neg lects' the. girl . ^ 
"latent issues", 



Ill, Analysis (Perceive elements of problem situation as 

view an event from multiple perspectives)!. , 



••Hi 



.5 



4 



Medium . 



Low 




Only one perspective recog- 
nized £ Answerb&ed^ 



From the girl's perspective: -Answer reflects awareness ; of. 
' ~ denia1"ofnrape~to^rotect^~ only two perspectives or one 
, • family or personal honor; . perspective en^hasi2ed:more. 
; obedience to the father. . : than the others . • Simple 
/ -Father's perspective: guilt analysis , but "on thd right 

i about not living up to oil- 'track". May be vagueness 
. tural norms regarding pro- about one of .the perspectives, ing of cultural issues, •. 
' tection of daughter; shame 1 c — — , ~- i -- Ji -- - ^ 1 n ' ' ' • ' ' ' ' 



^Jbiaselp, 
diced, lacks any understand- 



regarding daughter; wants \ issues, 
to manage crisis,, privately, \ ; .-' : '-\ 
Police's perspective: social • 
control need to see justice 
served. All three perspec- r /•• 
tives recognized and plausibly 
correct. ... 



given (eg. "she fears she's ,.. 
pregnant and doesn't wjnt to 
.exam;;or"kriS 
and is afraid to teTr or ' 
"fears additional attacks". 



IV. Synthesis (Formulate a variety of discretely plausible courses of action) 



5 



Medium 
3 



2 



Low 



At least 3 to 5 plausible 
and discrete courses of 
action outlined. Each 
/clearly presented without 
bias. Answer-reflects 
that the option selected 
in PART I was not the ... 
only possible one. 



Three to five options pre- 
sented but with an obvious 
bias toward one of the op- 
tions. Does not reflect a - 
"freeing up" of the mind to 
entertain other alternatives.- 
Imbalance in the clarity of 
options. Options are not 
discrete,' but are restate- 
ments of two or three options,.'. 



Only one real option listed. 
Option implausible or'unclear, 1 
Question hot answered (e.g. 
gives strategy for ' how one 
course of action might be 
implemented) . Illogical "Put 
attacker, girl and father in 
one room and see what 
happens". 



>/.lV..-v 



•■ . lit 



Hlqh 



Three plus two values ideh 
tif ied for each role 
Girls: obed i ence , 




Low 

m 



chasity. Fathers:; - control, 



Most obvious values Identified v Little or no awareness of 
(3) but answer 7 reflects laclc of— — what-values are, e.g, ."The 7 
understanding, of cultural and ' girl did what; she had; to 



Answer reflects, familiarity 
with Cultural: arid societal - 
values. ' : : lM^k!4W 



societal; values brought to bear or "she feels- sorry- for ; - ' . ; ; >; 
In tHis situation. ^/Emphasizes ' herself", Mistakes feelings - ; ^ *i 
only the fathejr *s>;0r daughter's for values. Expresses Mi.ves ''. 
values! ; - ; ~ ' for values, eg "doesn'riant ,.,.';': ; 

to get involved". • ";,/ ,;;; ^ 

'■■ ■■:V>V. > l -•* 



'•• y. 



VI. Execution (Perceive logical means-ends relationships with sufficient detail) 



5 



4 



Description is such 
that someone else could 
« carry out action. There 
is a logical and clear pro- 

• gressfon of steps - a 
"flow chart" in words. 
Optional actions stated, 
sequential, and lead t to 

",: closure.. Aware of pre- 



Medium 



is 



general; the sequence of 
actions is broken;; impor- 
misiing; some 



cou 



d "muddle 



Low 

#31 



V '-f :■.: .*.;. ■v-r-.->- .■ .■;:"./>.. 



,';v ''."''■' : ^'';/n-^r , 'p'- 



Description not clear, | 
Vaguei i over simp Iff ied ; ;; : > 
Someone would A not 
to imp lemen t so 1 u t ion 
the description of events . C #: J; 
Trial and error approach : : a : 
to implementation, 
does not lead to a solution- • ; > r 
only an array of activities, 
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V'.i.:,';Vo- 



C ;i! -v.;;v.V„-;, !.:(,, ,.. 



APPENDIX III 
The Scenarios* 



Cuban Crisis (I) 

As a case-worker for' the Department of Social Services, you have been 
assigned to work' with a young Cuban woman, found beaten and semi-conscious 
in a Cuban neighborhood. Taken by police to a hospital, she acknowledged 
that she had been sexually assaulted, but refused to identify her attacker; 
she became hysterical at the suggestion that she undergo a physical exami- 
nation to determine if rape had occurred. Her family has refused to 
•cooperate with police attempts to investigate the presumed rape, although 
the police have been able to determine her attacker's identity by infor- ; . 
mants in the Cuban community . The girT 's father has flatly refused to 
swear out a warrant , and the girl refused to acknowledge either that a rape 
has occurred or her assailant's identity.' Frustrated, the police have 
called you to come to the police station and take over the casey 



Land Development (II) 

You are a County Commissioner and a special meeting, has been called in 
order to settle a dispute that has occurred concerning the proposed deve- 
lopment of 100 acres of land along, a primitive river bank. The land in 
question is^25-feet beyond a marshy area that borders the river. ; The deve- 
loper plans to subdivide the land into brie acre residential tracts. The 
site is in a rural county that lies just 7 miles south of a rapidly growing 
metropolitan area. The river and surrounding areas are noted for their 
excellent fishing and camping, annually attracting- sportsmen and campers 
statewide and beyond. A major controversy has been raging ever since the 
developers intentions were made public. • Local environmental groups here 
.opposed the development while local businessmen have stressed the need for 
such a development. The developer had promised ;to delay the actual 
construct i on until envi rohmental groups completed an envi rorimental impact ; 
study . The special meeting has been called because several days ago the 
developer began construction of a road along the river bank and i. as also.., 
begun digging wells and septic tanks. The local environmental groups are 
demanding an immediate halt to construction. The developer claims that the 
construction delay is costing him thousands daily and that he intends to 
proceed with construction until concrete evidence is provided that the sub- 
division would be other than beneficiaT to the community. 
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Token Economy ( III) 

You are a School Board member in 7 an inner city school district. 
Disciplinary problems in the 3rd through 6th grades have tripled in the 
past year. The School Board* has r^ceivedjiumerous demands from parents to 
do something to restore order toyuie schoolrooms. Because of understaffing 
the classrooms are overcrowded/and it has become, increasingly more dif-.. 
ficult for teachers to'Tfiarrage^he students. Fighting, practical joking,., 
i nattent i veness , the use of vulgar 1 anguage and cheating abound. Several _ 
principals have asked the School Board to institute a token economy in all 
elementary school classrooms within the school district. Since this pro- 
posed solution was introduced at the last board meeting a controversy has 
raged between opposing factions in the P. T.O. Those favoring the institu- 
tion of a token economy claim it is the most efficient way to eliminate- v 
undesirable classroom behavior. Those opposing, the use of a token economy 
do lb primari ly on ethical and' moral grounds. The board meets next month 
to. make recommendations for further courses of action. - > ^ 



P 



